Enhancing NSCLC recurrence prediction with PET/CT habitat imaging, ctDNA, and integrative radiogenomics-blood insights.

While we recognize the prognostic importance of clinicopathological measures and circulating tumor DNA (ctDNA), the independent contribution of quantitative image markers to prognosis in non-small cell lung cancer (NSCLC) remains underexplored. In our multi-institutional study of 394 NSCLC patients, we utilize pre-treatment computed tomography (CT) and 18F-fluorodeoxyglucose positron emission tomography (FDG-PET) to establish a habitat imaging framework for assessing regional heterogeneity within individual tumors. This framework identifies three PET/CT subtypes, which maintain prognostic value after adjusting for clinicopathologic risk factors including tumor volume. Additionally, these subtypes complement ctDNA in predicting disease recurrence. Radiogenomics analysis unveil the molecular underpinnings of these imaging subtypes, highlighting downregulation in interferon alpha and gamma pathways in the high-risk subtype. In summary, our study demonstrates that these habitat imaging subtypes effectively stratify NSCLC patients based on their risk levels for disease recurrence after initial curative surgery or radiotherapy, providing valuable insights for personalized treatment approaches.

Single Stage Adaptive Multi-Attention Network for Image Restoration.

Recently attention-based networks have been successful for image restoration tasks. However, existing methods are either computationally expensive or have limited receptive fields, adding constraints to the model. They are also less resilient in spatial and contextual aspects and lack pixel-to-pixel correspondence, which may degrade feature representations. In this paper, we propose a novel and computationally efficient architecture Single Stage Adaptive Multi-Attention Network (SSAMAN) for image restoration tasks, particularly for image denoising and image deblurring. SSAMAN efficiently addresses computational challenges and expands receptive fields, enhancing robustness in spatial and contextual feature representation. Its Adaptive Multi-Attention Module (AMAM), which consists of Adaptive Pixel Attention Branch (APAB) and an Adaptive Channel Attention Branch (ACAB), uniquely integrates channel and pixel-wise dimensions, significantly improving sensitivity to edges, shapes, and textures. We perform extensive experiments and ablation studies to validate the performance of SSAMAN. Our model shows state-of-the-art results on various benchmarks, for example, on image denoising tasks, SSAMAN achieves a notable 40.08 dB PSNR on SIDD dataset, outperforming Restormer by 0.06 dB PSNR, with 41.02% less computational cost, and achieves a 40.05 dB PSNR on the DND dataset. For image deblurring, SSAMAN achieves 33.53 dB PSNR on GoPro dataset. Code and models are available at Github.

Synthetic PET from CT improves diagnosis and prognosis for lung cancer: Proof of concept.

[18F]Fluorodeoxyglucose positron emission tomography (FDG-PET) and computed tomography (CT) are indispensable components in modern medicine. Although PET can provide additional diagnostic value, it is costly and not universally accessible, particularly in low-income countries. To bridge this gap, we have developed a conditional generative adversarial network pipeline that can produce FDG-PET from diagnostic CT scans based on multi-center multi-modal lung cancer datasets (n = 1,478). Synthetic PET images are validated across imaging, biological, and clinical aspects. Radiologists confirm comparable imaging quality and tumor contrast between synthetic and actual PET scans. Radiogenomics analysis further proves that the dysregulated cancer hallmark pathways of synthetic PET are consistent with actual PET. We also demonstrate the clinical values of synthetic PET in improving lung cancer diagnosis, staging, risk prediction, and prognosis. Taken together, this proof-of-concept study testifies to the feasibility of applying deep learning to obtain high-fidelity PET translated from CT.

Automated Cellular-Level Dual Global Fusion of Whole-Slide Imaging for Lung Adenocarcinoma Prognosis.

Histopathologic whole-slide images (WSI) are generally considered the gold standard for cancer diagnosis and prognosis. Survival prediction based on WSI has recently attracted substantial attention. Nevertheless, it remains a central challenge owing to the inherent difficulties of predicting patient prognosis and effectively extracting informative survival-specific representations from WSI with highly compounded gigapixels. In this study, we present a fully automated cellular-level dual global fusion pipeline for survival prediction. Specifically, the proposed method first describes the composition of different cell populations on WSI. Then, it generates dimension-reduced WSI-embedded maps, allowing for efficient investigation of the tumor microenvironment. In addition, we introduce a novel dual global fusion network to incorporate global and inter-patch features of cell distribution, which enables the sufficient fusion of different types and locations of cells. We further validate the proposed pipeline using The Cancer Genome Atlas lung adenocarcinoma dataset. Our model achieves a C-index of 0.675 (±0.05) in the five-fold cross-validation setting and surpasses comparable methods. Further, we extensively analyze embedded map features and survival probabilities. These experimental results manifest the potential of our proposed pipeline for applications using WSI in lung adenocarcinoma and other malignancies.

Pathomic Features Reveal Immune and Molecular Evolution From Lung Preneoplasia to Invasive Adenocarcinoma.

Recent statistics on lung cancer, including the steady decline of advanced diseases and the dramatically increasing detection of early-stage diseases and indeterminate pulmonary nodules, mark the significance of a comprehensive understanding of early lung carcinogenesis. Lung adenocarcinoma (ADC) is the most common histologic subtype of lung cancer, and atypical adenomatous hyperplasia is the only recognized preneoplasia to ADC, which may progress to adenocarcinoma in situ (AIS) and minimally invasive adenocarcinoma (MIA) and eventually to invasive ADC. Although molecular evolution during early lung carcinogenesis has been explored in recent years, the progress has been significantly hindered, largely due to insufficient materials from ADC precursors. Here, we employed state-of-the-art deep learning and artificial intelligence techniques to robustly segment and recognize cells on routinely used hematoxylin and eosin histopathology images and extracted 9 biology-relevant pathomic features to decode lung preneoplasia evolution. We analyzed 3 distinct cohorts (Japan, China, and United States) covering 98 patients, 162 slides, and 669 regions of interest, including 143 normal, 129 atypical adenomatous hyperplasia, 94 AIS, 98 MIA, and 205 ADC. Extracted pathomic features revealed progressive increase of atypical epithelial cells and progressive decrease of lymphocytic cells from normal to AAH, AIS, MIA, and ADC, consistent with the results from tissue-consuming and expensive molecular/immune profiling. Furthermore, pathomics analysis manifested progressively increasing cellular intratumor heterogeneity along with the evolution from normal lung to invasive ADC. These findings demonstrated the feasibility and substantial potential of pathomics in studying lung cancer carcinogenesis directly from the low-cost routine hematoxylin and eosin staining.

Predicting benefit from immune checkpoint inhibitors in patients with non-small-cell lung cancer by CT-based ensemble deep learning: a retrospective study.

BACKGROUND: Only around 20-30% of patients with non-small-cell lung cancer (NCSLC) have durable benefit from immune-checkpoint inhibitors. Although tissue-based biomarkers (eg, PD-L1) are limited by suboptimal performance, tissue availability, and tumour heterogeneity, radiographic images might holistically capture the underlying cancer biology. We aimed to investigate the application of deep learning on chest CT scans to derive an imaging signature of response to immune checkpoint inhibitors and evaluate its added value in the clinical context. METHODS: In this retrospective modelling study, 976 patients with metastatic, EGFR/ALK negative NSCLC treated with immune checkpoint inhibitors at MD Anderson and Stanford were enrolled from Jan 1, 2014, to Feb 29, 2020. We built and tested an ensemble deep learning model on pretreatment CTs (Deep-CT) to predict overall survival and progression-free survival after treatment with immune checkpoint inhibitors. We also evaluated the added predictive value of the Deep-CT model in the context of existing clinicopathological and radiological metrics. FINDINGS: Our Deep-CT model demonstrated robust stratification of patient survival of the MD Anderson testing set, which was validated in the external Stanford set. The performance of the Deep-CT model remained significant on subgroup analyses stratified by PD-L1, histology, age, sex, and race. In univariate analysis, Deep-CT outperformed the conventional risk factors, including histology, smoking status, and PD-L1 expression, and remained an independent predictor after multivariate adjustment. Integrating the Deep-CT model with conventional risk factors demonstrated significantly improved prediction performance, with overall survival C-index increases from 0·70 (clinical model) to 0·75 (composite model) during testing. On the other hand, the deep learning risk scores correlated with some radiomics features, but radiomics alone could not reach the performance level of deep learning, indicating that the deep learning model effectively captured additional imaging patterns beyond known radiomics features. INTERPRETATION: This proof-of-concept study shows that automated profiling of radiographic scans through deep learning can provide orthogonal information independent of existing clinicopathological biomarkers, bringing the goal of precision immunotherapy for patients with NSCLC closer. FUNDING: National Institutes of Health, Mark Foundation Damon Runyon Foundation Physician Scientist Award, MD Anderson Strategic Initiative Development Program, MD Anderson Lung Moon Shot Program, Andrea Mugnaini, and Edward L C Smith.

Efficacy and clinicogenomic correlates of response to immune checkpoint inhibitors alone or with chemotherapy in non-small cell lung cancer.

The role of combination chemotherapy with immune checkpoint inhibitors (ICI) (ICI-chemo) over ICI monotherapy (ICI-mono) in non-small cell lung cancer (NSCLC) remains underexplored. In this retrospective study of 1133 NSCLC patients, treatment with ICI-mono vs ICI-chemo associate with higher rates of early progression, but similar long-term progression-free and overall survival. Sequential vs concurrent ICI and chemotherapy have similar long-term survival, suggesting no synergism from combination therapy. Integrative modeling identified PD-L1, disease burden (Stage IVb; liver metastases), and STK11 and JAK2 alterations as features associate with a higher likelihood of early progression on ICI-mono. CDKN2A alterations associate with worse long-term outcomes in ICI-chemo patients. These results are validated in independent external (n = 89) and internal (n = 393) cohorts. This real-world study suggests that ICI-chemo may protect against early progression but does not influence overall survival, and nominates features that identify those patients at risk for early progression who may maximally benefit from ICI-chemo.

Artificial Intelligence in Digital Pathology to Advance Cancer Immunotherapy.

Immune-checkpoint inhibitors (ICIs) have revolutionized the treatment of many malignancies. For instance, in lung cancer, however, only 20~30% of patients can achieve durable clinical benefits from ICI monotherapy. Histopathologic and molecular features such as histological type, PD-L1 expression, and tumor mutation burden (TMB), play a paramount role in selecting appropriate regimens for cancer treatment in the era of immunotherapy. Unfortunately, none of the existing features are exclusive predictive biomarkers. Thus, there is an imperative need to pinpoint more effective biomarkers to identify patients who may achieve the most benefit from ICIs. The adoption of digital pathology in clinical flow, as being powered by artificial intelligence (AI) especially deep learning, has catalyzed the automated analysis of tissue slides. With the breakthrough of multiplex bioimaging technology, researchers can comprehensively characterize the tumor microenvironment, including the different immune cells' distribution, function, and interaction. Here, we briefly summarize recent AI studies in digital pathology and share our perspective on emerging paradigms and directions to advance the development of immunotherapy biomarkers.

Limited Benefit from the Addition of Immunotherapy to Chemotherapy in TKI-Refractory EGFR-Mutant Lung Adenocarcinoma.

BACKGROUND: The benefit of chemotherapy combined with immunotherapy in EGFR-mutant lung adenocarcinoma (LUAD) patients whose tumor developed resistance to EGFR tyrosine kinase inhibitors (TKIs) is not thoroughly investigated. The goal of this retrospective cohort study is to assess the clinical efficiency of immunotherapy alone or in combination with chemotherapy in a real-world setting. METHODS: This retrospective cohort study enrolled LUAD patients with EGFR sensitive mutations whose tumor had acquired resistance to EGFR TKIs and received systemic treatment with chemotherapy (chemo; n = 84), chemotherapy combined with immunotherapy (chemoIO; n = 30), chemotherapy plus bevacizumab with or without IO (withBev; n = 42), and IO monotherapy (IO-mono; n = 22). Clinical progression-free survival (PFS) and overall survival (OS) were evaluated. Associations of clinical characteristics with outcomes were assessed using univariable and multi-covariate Cox Proportional Hazards regression models. RESULTS: A total of 178 patients (median age = 63.3; 57.9% females) with a median follow-up time of 42.0 (Interquartile range: 22.9-67.8) months were enrolled. There was no significant difference in PFS between chemoIO vs. chemo groups (5.3 vs. 4.8 months, p = 0.8). Compared to the chemo group, patients who received withBev therapy trended towards better PFS (6.1 months vs. 4.8; p = 0.3; HR 0.79; 95% CI: 0.52-1.20), while patients treated with IO-mono had inferior PFS (2.2 months; p = 0.001; HR 2.22; 95% CI: 1.37-3.59). Furthermore, PD-L1 level was not associated with PFS benefit in the chemoIO group. Patients with EGFR-mutant LUAD with high PD-L1 (≥50%) had shorter PFS (5.8 months) than non-EGFR/ALK LUAD patients who received chemoIO (12.8 months, p = 0.002; HR 0.22; 95% CI: 0.08-0.56) as first-line treatment. Chemotherapy-based therapy rendered similar benefit to patients with either EGFR exon19 deletion vs. L858R in the LUAD. CONCLUSIONS: This retrospective analysis revealed that immunotherapy provided limited additional benefit to chemotherapy in TKI-refractory EGFR-mutant LUAD. Chemotherapy alone or combined with bevacizumab remain good choices for patients with actionable EGFR mutations.

Chronic Lymphocytic Leukemia Progression Diagnosis with Intrinsic Cellular Patterns via Unsupervised Clustering.

Identifying the progression of chronic lymphocytic leukemia (CLL) to accelerated CLL (aCLL) or transformation to diffuse large B-cell lymphoma (Richter transformation; RT) has significant clinical implications as it prompts a major change in patient management. However, the differentiation between these disease phases may be challenging in routine practice. Unsupervised learning has gained increased attention because of its substantial potential in data intrinsic pattern discovery. Here, we demonstrate that cellular feature engineering, identifying cellular phenotypes via unsupervised clustering, provides the most robust analytic performance in analyzing digitized pathology slides (accuracy = 0.925, AUC = 0.978) when compared to alternative approaches, such as mixed features, supervised features, unsupervised/mixed/supervised feature fusion and selection, as well as patch-based convolutional neural network (CNN) feature extraction. We further validate the reproducibility and robustness of unsupervised feature extraction via stability and repeated splitting analysis, supporting its utility as a diagnostic aid in identifying CLL patients with histologic evidence of disease progression. The outcome of this study serves as proof of principle using an unsupervised machine learning scheme to enhance the diagnostic accuracy of the heterogeneous histology patterns that pathologists might not easily see.

Artificial intelligence-assisted mapping of proliferation centers allows the distinction of accelerated phase from large cell transformation in chronic lymphocytic leukemia.

Chronic lymphocytic leukemia/small lymphocytic lymphoma (CLL) is characterized morphologically by numerous small lymphocytes and pale nodules composed of prolymphocytes and paraimmunoblasts known as proliferation centers (PCs). Patients with CLL can undergo transformation to a more aggressive lymphoma, most often diffuse large B-cell lymphoma (DLBCL), known as Richter transformation (RT). An accelerated phase of CLL (aCLL) also may be observed which correlates with subsequent transformation to DLBCL, and may represent an early stage of transformation. Distinguishing PCs in CLL from aCLL or RT can be diagnostically challenging, particularly in small needle biopsy specimens. Available guidelines pertaining to distinguishing CLL from its' progressive forms are limited, subject to the morphologist's experience and are often not completely helpful in the assessment of scant biopsy specimens. To objectively assess the extent of PCs in aCLL and RT, and enhance diagnostic accuracy, we sought to design an artificial intelligence (AI)-based tool to identify and delineate PCs based on feature analysis of the combined individual nuclear size and intensity, designated here as the heat value. Using the mean heat value from the generated heat value image of all cases, we were able to reliably separate CLL, aCLL and RT with sensitive diagnostic predictive values.

Habitat Imaging Biomarkers for Diagnosis and Prognosis in Cancer Patients Infected with COVID-19.

OBJECTIVES: Cancer patients have worse outcomes from the COVID-19 infection and greater need for ventilator support and elevated mortality rates than the general population. However, previous artificial intelligence (AI) studies focused on patients without cancer to develop diagnosis and severity prediction models. Little is known about how the AI models perform in cancer patients. In this study, we aim to develop a computational framework for COVID-19 diagnosis and severity prediction particularly in a cancer population and further compare it head-to-head to a general population. METHODS: We have enrolled multi-center international cohorts with 531 CT scans from 502 general patients and 420 CT scans from 414 cancer patients. In particular, the habitat imaging pipeline was developed to quantify the complex infection patterns by partitioning the whole lung regions into phenotypically different subregions. Subsequently, various machine learning models nested with feature selection were built for COVID-19 detection and severity prediction. RESULTS: These models showed almost perfect performance in COVID-19 infection diagnosis and predicting its severity during cross validation. Our analysis revealed that models built separately on the cancer population performed significantly better than those built on the general population and locked to test on the cancer population. This may be because of the significant difference among the habitat features across the two different cohorts. CONCLUSIONS: Taken together, our habitat imaging analysis as a proof-of-concept study has highlighted the unique radiologic features of cancer patients and demonstrated effectiveness of CT-based machine learning model in informing COVID-19 management in the cancer population.

Artificial intelligence strategy integrating morphologic and architectural biomarkers provides robust diagnostic accuracy for disease progression in chronic lymphocytic leukemia.

Artificial intelligence-based tools designed to assist in the diagnosis of lymphoid neoplasms remain limited. The development of such tools can add value as a diagnostic aid in the evaluation of tissue samples involved by lymphoma. A common diagnostic question is the determination of chronic lymphocytic leukemia (CLL) progression to accelerated CLL (aCLL) or transformation to diffuse large B-cell lymphoma (Richter transformation; RT) in patients who develop progressive disease. The morphologic assessment of CLL, aCLL, and RT can be diagnostically challenging. Using established diagnostic criteria of CLL progression/transformation, we designed four artificial intelligence-constructed biomarkers based on cytologic (nuclear size and nuclear intensity) and architectural (cellular density and cell to nearest-neighbor distance) features. We analyzed the predictive value of implementing these biomarkers individually and then in an iterative sequential manner to distinguish tissue samples with CLL, aCLL, and RT. Our model, based on these four morphologic biomarker attributes, achieved a robust analytic accuracy. This study suggests that biomarkers identified using artificial intelligence-based tools can be used to assist in the diagnostic evaluation of tissue samples from patients with CLL who develop aggressive disease features. © 2021 The Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.

CellSpatialGraph: Integrate hierarchical phenotyping and graph modeling to characterize spatial architecture in tumor microenvironment on digital pathology.

We present CellSpatialGraph, an integrated clustering and graph-based framework, to investigate the cellular spatial structure. Due to the lack of a clear understanding of the cell subtypes in the tumor microenvironment, unsupervised learning is applied to uncover cell phenotypes. Then, we build local cell graphs, referred to as supercells, to model the cell-to-cell relationships at a local scale. After that, we apply clustering again to identify the subtypes of supercells. In the end, we build a global graph to summarize supercell-to-supercell interactions, from which we extract features to classify different disease subtypes.

Automatic Whole Slide Pathology Image Diagnosis Framework via Unit Stochastic Selection and Attention Fusion.

Pathology tissue slides are taken as the gold standard for the diagnosis of most cancer diseases. Automatic pathology slide diagnosis is still a challenging task for researchers because of the high-resolution, significant morphological variation, and ambiguity between malignant and benign regions in whole slide images (WSIs). In this study, we introduce a general framework to automatically diagnose different types of WSIs via unit stochastic selection and attention fusion. For example, a unit can denote a patch in a histopathology slide or a cell in a cytopathology slide. To be specific, we first train a unit-level convolutional neural network (CNN) to perform two tasks: constructing feature extractors for the units and for estimating a unit's non-benign probability. Then we use our novel stochastic selection algorithm to choose a small subset of units that are most likely to be non-benign, referred to as the Units Of Interest (UOI), as determined by the CNN. Next, we use the attention mechanism to fuse the representations of the UOI to form a fixed-length descriptor for the WSI's diagnosis. We evaluate the proposed framework on three datasets: histological thyroid frozen sections, histological colonoscopy tissue slides, and cytological cervical pap smear slides. The framework achieves diagnosis accuracies higher than 0.8 and AUC values higher than 0.85 in all three applications. Experiments demonstrate the generality and effectiveness of the proposed framework and its potentiality for clinical applications.

Text-Guided Neural Network Training for Image Recognition in Natural Scenes and Medicine.

Convolutional neural networks (CNNs) are widely recognized as the foundation for machine vision systems. The conventional rule of teaching CNNs to understand images requires training images with human annotated labels, without any additional instructions. In this article, we look into a new scope and explore the guidance from text for neural network training. We present two versions of attention mechanisms to facilitate interactions between visual and semantic information and encourage CNNs to effectively distill visual features by leveraging semantic features. In contrast to dedicated text-image joint embedding methods, our method realizes asynchronous training and inference behavior: a trained model can classify images, irrespective of the text availability. This characteristic substantially improves the model scalability to multiple (multimodal) vision tasks. We also apply the proposed method onto medical imaging, which learns from richer clinical knowledge and achieves attention-based interpretable decision-making. With comprehensive validation on two natural and two medical datasets, we demonstrate that our method can effectively make use of semantic knowledge to improve CNN performance. Our method performs substantial improvement on medical image datasets. Meanwhile, it achieves promising performance for multi-label image classification and caption-image retrieval as well as excellent performance for phrase-based and multi-object localization on public benchmarks.

Loss-Based Attention for Interpreting Image-Level Prediction of Convolutional Neural Networks.

Although deep neural networks have achieved great success on numerous large-scale tasks, poor interpretability is still a notorious obstacle for practical applications. In this paper, we propose a novel and general attention mechanism, loss-based attention, upon which we modify deep neural networks to mine significant image patches for explaining which parts determine the image decision-making. This is inspired by the fact that some patches contain significant objects or their parts for image-level decision. Unlike previous attention mechanisms that adopt different layers and parameters to learn weights and image prediction, the proposed loss-based attention mechanism mines significant patches by utilizing the same parameters to learn patch weights and logits (class vectors), and image prediction simultaneously, so as to connect the attention mechanism with the loss function for boosting the patch precision and recall. Additionally, different from previous popular networks that utilize max-pooling or stride operations in convolutional layers without considering the spatial relationship of features, the modified deep architectures first remove them to preserve the spatial relationship of image patches and greatly reduce their dependencies, and then add two convolutional or capsule layers to extract their features. With the learned patch weights, the image-level decision of the modified deep architectures is the weighted sum on patches. Extensive experiments on large-scale benchmark databases demonstrate that the proposed architectures can obtain better or competitive performance to state-of-the-art baseline networks with better interpretability. The source codes are available on: https://github.com/xsshi2015/Loss-based-Attention-for-Interpreting-Image-level-Prediction-of-Convolutional-Neural-Networks.

Chromosome Segmentation via Data Simulation and Shape Learning.

Karyotyping, consisting of single chromosome segmentation and classification, is widely used in the cytogenetic analysis for chromosome abnormality detection. Many studies have reported automatic chromosome classification with high accuracy. Nevertheless, they usually require manual chromosome segmentation beforehand. There are two critical issues in automatic chromosome segmentation: 1) scarce annotated images for model training, and 2) multiple region combinations to form single chromosomes. In this study, two simulation strategies are proposed for training data argumentation to alleviate data scarcity. Besides, we present an optimization-based shape learning method to evaluate the shape of formed single chromosomes, which achieve the global minimum loss when segmented regions are correctly combined. Experiments on a public dataset demonstrate the effectiveness of the proposed method. The data simulation strategy has significantly increased the segmentation results by 15.8% and 46.3% of the Dice coefficient on non-overlapped and overlapped regions. Moreover, the proposed optimization-based method separates overlapped chromosomes with an accuracy of 96.2%.

Rule-based automatic diagnosis of thyroid nodules from intraoperative frozen sections using deep learning.

Frozen sections provide a basis for rapid intraoperative diagnosis that can guide surgery, but the diagnoses often challenge pathologists. Here we propose a rule-based system to differentiate thyroid nodules from intraoperative frozen sections using deep learning techniques. The proposed system consists of three components: (1) automatically locating tissue regions in the whole slide images (WSIs), (2) splitting located tissue regions into patches and classifying each patch into predefined categories using convolutional neural networks (CNN), and (3) integrating predictions of all patches to form the final diagnosis with a rule-based system. To be specific, we fine-tune the InceptionV3 model for thyroid patch classification by replacing the last fully connected layer with three outputs representing the patch's probabilities of being benign, uncertain, or malignant. Moreover, we design a rule-based protocol to integrate patches' predictions to form the final diagnosis, which provides interpretability for the proposed system. On 259 testing slides, the system correctly predicts 95.3% (61/64) of benign nodules and 96.7% (148/153) of malignant nodules, and classify 16.2% (42/259) slides as uncertain, including 19 benign and 16 malignant slides, which are a sufficiently small number to be manually examined by pathologists or fully processed through permanent sections. Besides, the system allows the localization of suspicious regions along with the diagnosis. A typical whole slide image, with 80, 000 × 60, 000 pixels, can be diagnosed within 1 min, thus satisfying the time requirement for intraoperative diagnosis. To the best of our knowledge, this is the first study to apply deep learning to diagnose thyroid nodules from intraoperative frozen sections. The code is released at https://github.com/PingjunChen/ThyroidRule.

Interactive thyroid whole slide image diagnostic system using deep representation.

BACKGROUND AND OBJECTIVES: The vast size of the histopathology whole slide image poses formidable challenges to its automatic diagnosis. With the goal of computer-aided diagnosis and the insights that suspicious regions are generally easy to identify in thyroid whole slide images (WSIs), we develop an interactive whole slide diagnostic system for thyroid frozen sections based on the suspicious regions preselected by pathologists. METHODS: We propose to generate feature representations for the suspicious regions via extracting and fusing patch features using deep neural networks. We then evaluate region classification and retrieval on four classifiers and three supervised hashing methods based on the feature representations. The code is released at https://github.com/PingjunChen/ThyroidInteractive. RESULTS: We evaluate the proposed system on 345 thyroid frozen sections and achieve 96.1% cross-validated classification accuracy, and retrieval mean average precision (MAP) of 0.972. CONCLUSIONS: With the participation of pathologists, the system possesses the following four notable advantages compared to directly handling whole slide images: 1) Reduced interference of irrelevant regions; 2) Alleviated computation and memory cost. 3) Fine-grained and precise suspicious region retrieval. 4) Cooperative relationship between pathologists and the diagnostic system. Additionally, experimental results demonstrate the potential of the proposed system on the practical thyroid frozen section diagnosis.